Methods & Results for Behavioural Study of Moral Judgements of Humans vs. AI

▼

Methods

Participants

Number of participants at start: 93

Number of participants after cleaning: 90

Participants removed during data cleaning timed out.

N = 90

M_age = 46.63 , SD_age = 14.75

Age range = 20 – 81 years

Gender Composition of Sample

Gender	n	%
Man	38	42.2
Non-binary	2	2.2
Prefer Not To Say	1	1.1
Woman	49	54.4

Ethnic / Cultural Identities

Because the ethnicity question used UK-specific categories but the sample was US-based, free-text responses were cleaned and recoded into broader categories based on standard US Census categories, with an additional category for those who identified with multiple ethnicities/cultural identities. The table below summarises the cleaned and recoded responses.

Appendix A: Table Summarising Participants’ Cultural Identities

US Census Ethnic/Cultural Categories	n	%
White	58	58 (64.4%)
Black / African American	10	10 (11.1%)
Multiple Ethnic/Cultural Identities	9	9 (10%)
Asian	5	5 (5.6%)
Other	5	5 (5.6%)
Southwest Asian / North African	3	3 (3.3%)

Nationalities

For this study, the sample was fixed to US nationals & residents. Here I run a quick check to ensure that this was the case.

Number of participants who identified as US nationals:

90 out of 90 participants

Number of participants who stated their country of residence is the United States:

90 out of 90 participants

Nationality Verification
• US nationals: 90 / 90 (Total Sample)
• US residents: 90 / 90 (Total Sample)

Results

Main Analysis (All Data)

First, the GLMM model was fit with a full random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=df, family = binomial(link = "logit"),
               control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 1. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 2. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots

Interaction Effects Plots

Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means of PDE within PBH

Table 3. Estimated marginal means by PBH × PDE within Agent

Table 4. Estimated marginal means by PDE x Agent within PBH

Table 5. Estimated marginal means by Agent × PDE within PDE

Contrasts of EMMs

Table 8. Pairwise contrasts for PDE within PBH

[1] “
”

Table 6. Pairwise contrasts for PBH × PDE within Agent

Table 7. Pairwise contrasts for PDE × Agent within PBH

Table 8. Pairwise contrasts for PBH × Agent within PDE

Two Sided Equivalence Test for Main Analysis

Table 9. TOST-test for Practical Equivalence for Main Model

Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.

Model Comparisons

Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.

Model Checks

Removing dilemmas with ambiguous MG predictions

The additional dilemmas developed in-lab did not have explicit predictions in the MG model. For the most part, these could be estimated based off of the predictions made for alternative dilemmas, and/or guided by the fulfilments of PBH/PDE. However, the unprohibited better alternative equipment-only disproportionate cost intended and the unprohibited better alternative disproportionate cost scenarios had particularly ambiguous estimates. Therefore, for the first study and this study, I conduct analyses with, and without, these dilemmas. Until, at least there is sufficient information available to provide more appropriate predictions.

First, the GLMM model was fit with a full random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=reduced_df, family = binomial(link = "logit"),
                   control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE+Agent|ID) + (1|Dilemma),  data=reduced_df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This model returned no singularity error but the correlations between the RE slopes and the intercept were very high (0.82-0.95), suggesting overfitting. Therefore, the random effects structure was further reduced by removing the random slope for Agent. This returned no singularity warning and much lower correlations between the RE slopes and intercept.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=reduced_df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots

Interaction Effects Plots

Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH

Table 3. Estimated marginal means by PBH × PDE within Agent

Table 4. Estimated marginal means by PDE x Agent within PBH

Table 5. Estimated marginal means by Agent × PBH within PDE

Contrasts of EMMs

Table 8. Pairwise contrasts for PDE within PBH

Table 6. Pairwise contrasts for PBH × PDE within Agent

Table 7. Pairwise contrasts for PDE × Agent within PBH

Table 8. Pairwise contrasts for PBH × Agent within PDE

Two Sided Equivalence Test for Main Analysis (Reduced Dilemmas)

Table 9. TOST-test for Practical Equivalence for Model with Reduced Dilemmas

Figure 9.
TOST results for Analysis of Model with Reduced Dilemmas

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.

Model Comparisons

Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.

Model Checks

Removing Participants who failed a costless rescue Human scenario

First, the GLMM model was fit with a match random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=df_match, family = binomial(link = "logit"),
                      control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was match (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=df_match, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots

Interaction Effects Plots

Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH

Table 3. Estimated marginal means by PBH × PDE within Agent

Table 4. Estimated marginal means by PDE x Agent within PBH

Table 5. Estimated marginal means by Agent × PBH within PDE

Contrasts of EMMs

Table 7. Pairwise contrasts for PDE within PBH

Table 6. Pairwise contrasts for PBH × PDE within Agent

Table 7. Pairwise contrasts for PDE × Agent within PBH

Table 8. Pairwise contrasts for PBH × Agent within PDE

Two Sided Equivalence Test for Main Analysis

Table 9. TOST-test for Practical Equivalence for Main Model

Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.

Model Comparisons

Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.

Model Checks

Removing Participants who failed CR Human scenario and dilemmas with ambiguous MG predictions.

First, the GLMM model was fit with a reduced_match random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=reduced_df_match, family = binomial(link = "logit"),
                      control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced_match (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=reduced_df_match, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots

Interaction Effects Plots

Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH

Table 3. Estimated marginal means by PBH × PDE within Agent

Table 4. Estimated marginal means by PDE x Agent within PBH

Table 5. Estimated marginal means by PBH within PDE

Contrasts of EMMs

Table 7. Pairwise contrasts for PDE within PBH

Table 6. Pairwise contrasts for PBH × PDE within Agent

Table 7. Pairwise contrasts for PDE × Agent within PBH

Table 8. Pairwise contrasts for PBH × Agent within PDE

Two Sided Equivalence Test for Main Analysis

Table 9. TOST-test for Practical Equivalence for Main Model

Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.

Model Comparisons

Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.

Model Checks